1. Field of the Invention
This invention pertains in general to protecting a computer from malicious software and in particular to techniques for intercepting system calls.
2. Description of the Related Art
Typically, a process running in an operating system has limited access to system resources to prevent the process from maliciously or accidentally interfering with the functioning of the operating system and other processes. System calls are provided to processes by the operating system as a way for the process to access system resources when necessary. For example, a process may not be allowed direct access to a hard disk, but read( ) and write( ) system calls may be provided to give the process controlled access to the disk. Typically, when a process makes a system call, control is transferred to the operating system kernel which carries out the system call and then returns control to the process.
Malicious software can use system calls to cause undesirable effects on the system. Malicious software running in a process at a user privilege level may make improper system calls in an attempt to switch to the supervisor privilege level and have unrestricted access to the system. Also, malicious software running at any privilege level may make system calls to perform unwanted functions, such as erasing important files on a disk or propagating a virus over the Internet. The malicious software may take advantage of security flaws present in the system calls such as the inability of the system call to properly handle unexpected parameters. Alternatively, the malicious software may use the calls as intended by the operating system but for malicious purposes.
Security software can monitor system calls for evidence of malicious activity and take remedial action if necessary. Kernel patching is a technique for allowing security software to monitor system calls. Kernel patching modifies code and data structures in the operating system kernel to call security software before performing the requested operating system function. For example, the write( ) system call, which is implemented in the kernel, may be patched so that it initially performs various security checks to ensure that it is not being called maliciously. The patching may insert new code at the start of the system call to examine the calling process and the parameters passed to the system call. If there is no security risk, the system call is allowed to continue execution normally, but if a security risk is detected, some action is taken, such as stopping the process and notifying the user.
Kernel patching is performed by legitimate security software, but it is also a technique employed by malicious software such as rootkits and Trojan horses in order to prevent detection and ensure survival. Though there are different ways of dealing with this security threat, some operating systems attempt to prohibit all kernel patching. Though this technique may prevent malicious software from patching the kernel, it also unfortunately prevents legitimate security software from doing so. Often, kernel patching is the best way to effectively and efficiently guard against certain types of attacks on the system.
One way that operating systems may prevent kernel patching is by having an operating system protection module that periodically scans the kernel code and data structures to make sure they have not been modified. If the operating system detects that the kernel has been modified, it may take some action such has halting the system and displaying an error message. An example of such an operating system protection module is PATCHGUARD on 64-bit versions of MICROSOFT WINDOWS operating systems. Although the operating system may provide application programming interface (API) functions as hooks for executing security code as an alternative to kernel patching, the API may not provide all necessary functionality and it may be used by malicious code, even if undocumented. Therefore, there is a need in the art for a way to provide enhanced security in a computer having an operating system that ostensibly prevents kernel patching.